This appendix introduces the basic idea of statistical hypothesis testing. It is a prerequisite for understanding the χ2 test and ANOVA F-test used in Chapter 06: Feature Selection. The goal here is not to teach statistics in detail. The goal is to understand what a statistical test is asking, what a p-value means, and how to interpret the final decision.
Machine learning often works with a dataset that contains a sample of observations rather than every possible observation in the population. Because a sample is only one random selection, its numbers will naturally change from sample to sample.
Imagine a fair coin. Under the assumption that the coin is fair, the probability of heads is 50%. Does every sample of 20 flips contain exactly 10 heads? No. Try generating samples and notice how much the observed percentage moves around even when the underlying probability stays at 50%.
Single sample
| Current sample | Heads | Observed percentage | Expected under H0 |
|---|---|---|---|
| — | — | — | 50% |
100 samples (distribution)
Each bar is one simulated sample; its height is the percentage of heads in that sample. The dashed red line marks 50% — the value expected under H0.
This is the basic difficulty of statistical inference: an observed difference does not automatically mean that the underlying population is different. Some difference can appear simply because of sampling variability.
Suppose we want to know whether two groups have different population means. We usually cannot measure the entire population, so we collect samples from the two groups and compare them.
| Term | Basic meaning | Example |
|---|---|---|
| Population | The complete group we are interested in. | All customers of a bank. |
| Sample | The observations actually collected. | 500 sampled customers. |
| Parameter | A numerical property of the population. | True population mean. |
| Statistic | A numerical property calculated from the sample. | Sample mean. |
The null hypothesis is the default assumption that we test against. In many introductory tests it represents no association, no difference, or no effect.
The alternative hypothesis describes what we would consider evidence for instead of H0.
Most hypothesis tests can be understood using the following five-step sequence:
A test statistic is a number calculated from the sample that summarizes how far the observed data depart from what H0 predicts.
Different tests use different statistics. For example:
| Test | Test statistic | Basic question |
|---|---|---|
| Chi-square | \(\chi^2\) | How far are observed counts from the counts expected under independence? |
| ANOVA | \(F\) | How large is between-group variation relative to within-group variation? |
The formula changes from one test to another, but the purpose is the same: turn the observed data into a number that can be compared with what is expected under H0.
This is the step students often skip mentally. We imagine that H0 is true, and then ask what values of the test statistic would commonly occur through random sampling.
Most results will be in the middle of the distribution. More extreme results occur less often.
The curve below represents the distribution of a generic standardized test statistic when H0 is true. Move the slider to place the observed result farther from the center.
| Observed distance | 1.0 |
|---|---|
| Approximate two-sided p-value | 0.317 |
| Decision at α = 0.05 | Do not reject H0 |
This visualization is intentionally generic. Chi-square and ANOVA do not use this exact bell-shaped null distribution; they have their own sampling distributions. Unlike this two-sided illustration, chi-square and F distributions are one-tailed: only unusually large values count as extreme, since both statistics are always non-negative. The important idea carries over regardless: more extreme results occupy less of the distribution under H0.
The phrase “at least as extreme” matters. We are not asking for the probability of obtaining exactly the same numerical result. We count the observed result and results that would be even more surprising under H0.
| p-value | Intuition |
|---|---|
| Large | The result is not especially unusual if H0 is true. |
| Small | The result would be unusual if H0 were true. |
Suppose an ANOVA test produces \(p = 0.003\). The correct interpretation at this level is:
We do not say that H0 has a 0.3% probability of being true.
Before making the test decision, we choose a significance level, commonly written as \(\alpha\). A common choice is \(\alpha = 0.05\). This is the same \(\alpha\) from your intro statistics course: the Type I error rate — the chance of rejecting H0 when H0 is actually true.
| Comparison | Decision | Basic meaning |
|---|---|---|
| \(p \le \alpha\) | Reject H0 | The observed result is sufficiently unusual under H0. |
| \(p > \alpha\) | Do not reject H0 | The evidence is not sufficiently strong to reject H0. |
| Incorrect interpretation | Why it is wrong | Better interpretation |
|---|---|---|
| “p = 0.03 means there is a 3% chance H0 is true.” | The p-value is calculated assuming H0, not by calculating the probability that H0 is true. | Assuming H0 is true, results this extreme or more extreme would be rare. |
| “p = 0.03 means the result has 97% accuracy.” | A p-value is not an accuracy measure. | The p-value measures how unusual the data are under H0. |
| “p > 0.05 proves there is no relationship.” | Failure to reject H0 is not proof that H0 is true. | The data do not provide sufficiently strong evidence against H0. |
| “Smaller p-value always means a more important feature.” | Statistical significance and practical / predictive importance are different ideas. | Use the test to assess evidence against H0; use effect size or predictive evaluation for other questions. |
With a very large sample, even a small difference can produce a very small p-value. With a small sample, a fairly large difference may fail to reach statistical significance.
| Step | Question |
|---|---|
| 1 | What is H0? |
| 2 | What would the data look like if H0 were true? |
| 3 | How far is the observed result from that expectation? |
| 4 | How often would a result this extreme occur under H0? |
| 5 | Is the resulting p-value small enough to reject H0 at the chosen α? |
For a categorical feature and categorical target, the chi-square test asks whether the two variables are independent.
| Part | Chi-square feature-selection interpretation |
|---|---|
| H0 | The feature and target are independent. |
| Observed data | The counts in the contingency table. |
| Expected data | The counts we would expect if the feature and target were independent. |
| Test statistic | \(\chi^2\), measuring how far observed counts are from expected counts. |
| p-value | How unusual this amount of departure would be if the variables were really independent. |
| Small p-value | Evidence against independence; the feature and target are associated. |
For a numerical feature and categorical target, ANOVA asks whether the group means are sufficiently different to provide evidence against the assumption that all population means are equal.
| Part | ANOVA feature-selection interpretation |
|---|---|
| H0 | All population group means are equal. |
| Observed data | The feature values within each target class. |
| Test statistic | \(F\), comparing between-group variability with within-group variability. |
| p-value | How unusual the observed separation among group means would be if the population means were really equal. |
| Small p-value | Evidence that at least one population mean differs from the others. |
Answer all 5 questions. Click an option for instant feedback.
Your score: 0 / 5